Search for: All records

Advancements in ultra-low-power tiny machine learning (TinyML) systems promise to unlock an entirely new class of smart applications. However, continued progress is limited by the lack of a widely accepted and easily reproducible benchmark for these systems. To meet this need, we present MLPerf Tiny, the first industry-standard benchmark suite for ultra-low-power tiny machine learning systems. The benchmark suite is the collaborative effort of more than 50 organizations from industry and academia and reflects the needs of the community. MLPerf Tiny measures the accuracy, latency, and energy of machine learning inference to properly evaluate the tradeoffs between systems. Additionally, MLPerf Tiny implements a modular design that enables benchmark submitters to show the benefits of their product, regardless of where it falls on the ML deployment stack, in a fair and reproducible manner. The suite features four benchmarks: keyword spotting, visual wake words, image classification, and anomaly detection. 

Scientific communities are increasingly adopting machine learning and deep learning models in their applications to accelerate scientific insights. High performance computing systems are pushing the frontiers of performance with a rich diversity of hardware resources and massive scale-out capabilities. There is a critical need to understand fair and effective benchmarking of machine learning applications that are representative of real-world scientific use cases. MLPerf ™ is a community-driven standard to benchmark machine learning workloads, focusing on end-to-end performance metrics. In this paper, we introduce MLPerf HPC, a benchmark suite of large-scale scientific machine learning training applications, driven by the MLCommons ™ Association. We present the results from the first submission round including a diverse set of some of the world’s largest HPC systems. We develop a systematic framework for their joint analysis and compare them in terms of data staging, algorithmic convergence and compute performance. As a result, we gain a quantitative understanding of optimizations on different subsystems such as staging and on-node loading of data, compute-unit utilization and communication scheduling enabling overall >10× (end-to-end) performance improvements through system scaling. Notably, our analysis shows a scale-dependent interplay between the dataset size, a system’s memory hierarchy and training convergence that underlines the importance of near-compute storage. To overcome the data-parallel scalability challenge at large batch-sizes, we discuss specific learning techniques and hybrid data-and-model parallelism that are effective on large systems. We conclude by characterizing each benchmark with respect to low-level memory, I/O and network behaviour to parameterize extended roofline performance models in future rounds. 

Abstract To feed the world population while mitigating pressing nitrogen (N) pollution problems, tremendous efforts have been devoted to developing and implementing N‐efficient technologies in crop or livestock production, but limited progress has been made. The N management improvement on a farm does not necessarily translate to N pollution reduction on a broader scale due to complex responses of natural and human systems and lack of coordination among stakeholders. Consequently, it is imperative to develop an N management framework that encompasses the complex N dynamics across systems and spatial scales, yet simple enough to guide policies and actions of various stakeholders. Here, we propose a new framework,CAFE, that defines four N management systems (Cropping,Animal‐crop,Food, andEcosystem) in a hierarchical manner, and apply it to 13 representative countries to partition N surpluses across systems in a simple and consistent manner, thereby facilitating the identification and prioritization of systems‐based intervention strategies. Surprisingly, theCropping system contributes less than half of the total N surplus within itsEcosystem for most countries, highlighting the importance of N management beyond croplands. This framework reveals that the relevant priorities and key stakeholders for enhanced N management vary among countries, such as improving theCropping‐system efficiencies in China, adjusting the animal‐crop portfolio in the Netherlands, reducing food wastage in the U.S., and lowering crop storage losses and increasing overall production capacities in African countries. As N surplus increases along theCAFEhierarchy, systems‐based intervention strategies are revealed: (a) coupling chemical fertilizers with other N sources by maintaining half of the N from manure and biological N fixation; (b) coupling animal‐crop production by reducing animal density to lower than 1.2 livestock units per hectare, and increasing self‐sufficiency of animal feed to above 50%; (c) coupling food trade with domestic demand and production; and (d) coupling population needs for economic opportunities with environmental capacity of the region. This novel framework can help unpack the “wicked” N management challenges across systems to provide new insights and tools for improving N management on and beyond farms.